United States Patent 

MacKay et al. 



US0051481S4A 

[ll] Patent Number 
[43] Date of Patent: 



5,148,154 
Sep. 15, 1992 



[54] MULTI-DIMENSIONAL USER INTERFACE 

[75] Inventors: Michael T. MacKay, Vallejo; Robert 
J. Berger, Menlo Park; Robert Duffy, 
MDpitasj Ted E, Longford, Fremont, 
all of Calif. 

[73] Assignee: Sony Corporation of America, Park 
Ridge, NJ... 

[21] Appl. No.: 622,821 

[22] Filed: Dec 4, 1990 

[51] Int. a> .... G09G 3/02 

[52] US. CL .„ _ 340/712; 340/729; 

340/721 

[58] Field of Search 340/729, 721, 712, 723; 

364/522. 521; 395/154, 159 

[56] References Cited 

U.S. PATENT DOCUMENTS 

Re. 32,632 3/19B8 Atkinson : 340/709 

4,533,910 8/1985 Sukonick « al 340/721 

4,355,775 11/1985 Pike 364/900 

4,622,545 I1/19B6 Atkinson , 340/747 

4,697,178 9/1987 Hcekd 364/522 

4,746,994 S/J988 Ettlinger „... 360/13 

4.748.618 5/1988 Brown et al. 370/94 

4,785.408 11/1988 Britton et aL _ 364/513 

4,812.834 3/1989 Wells ..... .„. 340/721 

4,847.604 7/1989 Doyle 340/706 

4,868.766 8/198'9 Oonerholt „ „ 364/S22 

4,879.751 11/1989 Franks et el .381/119 

4.884,223 11/1989 Ingle et al. 364/550 

4,899,136 2/1990 Beard et al 340/706 

4,914.732 4/1990 Henderson et aL 340/825 

4,931,7B3 6/1990 Atkinson 340/710 

4.935.B65 6/1990 Rowe et aL 364/188 

4,939,507 7/1990 Beard et ai 340/706 

4,994,989 2/1991 Usami et aL 340/729 

5,040,131 8/1991 Torres 364/521 



OTHER PUBLICATIONS 

Alexander, "Visualizing Cleared -Off Desktops" Com- 
puter World, May 6, 1991, p. 20. 
Video Tape "Sony AVTC Venues and View", Oct 
1990. 

Primary Examiner— Jeffery A. Brier 
Assistant Examiner — Jick Chin 

Attorney, Agent, or Firm — Blakely, Sokoloff, Taylor & 
Za fi nan 

[57] ABSTRACT 

The present invention provides apparatus and methods 
. for a multi-dim en s i onal user interface for use in audio 
visual production. A display system including a central 
processing unit (CPU) is coupled through appropriate 
input/output (I/O) circuitry to input devices, such as a 
keyboard, digital pad and/or track ball as well as a 
display device. The CPU is further coupled to a hard 
disk drive for the storage of programs and data, and is 
also coupled to a network through which the CPU may 
communicate with a variety of system resource devices 
such as editors, music synthesizers, graphics generators, 
scheduling resources, audio enhancement resources, 
etc. A user viewing the interface on the display may 
utilize one of the input devices, such as by way of exam* 
pie, the keyboard, to select, incorporate or otherwise 
integrate the various system resources to develop a 
unified multi-media production. The user interface of 
the present invention includes a control frame which in 
practice substantially fills all of the display screen of the 
display and is consistent for all user applications. The 
control frame is comprised of control panels which 
surround a variety of subwindows and- acts as a consis- 
tent control area for all users of the interface. Once 
defined, elements may be selectively placed on an event 
horizon bar in the control frame. The placement of an 
element on the event horizon results in the display of 
timing data for the element, relative to other elements 
on the event horizon. 

56 Claims, 7 Drawing Sheets 
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composers, and the like may provide both real time and 
MULTI-DIMENSIONAL USER INTERFACE non-real time input to exchange necessary project data, 

and effectively coordinate the production of the entire 
BACKGROUND OF THE INVENTION media work. Accordingly, one of the requirements of 

1 Field of the Invention S any common multi-media user interface is the ability to 

The present invention relates to apparatus and meth- integrate multi-media types, and to provide the operator 
ods for displaying, manipulating, and selecting multi- with the ability to manage large quantities of informa- 
media or computer stored information in a window tion in an understandable and efficient manner. The user 
environment of a computer display system. More par- interface most be intuitive and flexible to accommodate 
ticularly, the present invention relates to an improved 10 a variety, of operator editing styles and personalities, 
user interface to provide a unified operator interface for For example, a music composer who thinks typically in 
a wide range of systems which must be coordinated and terms of scores, notes and related music timing, should 
monitored in a multi-media production system. be able to work in that environment using a standard 

2. Art Background user interface, and not be required to work in terms of 

Videotape editing environments have evolved from " video time code or other non-music related external 
providing simple editing cots to the incorporation of standards. Similarly, the film production director or 
full featured graphics, film to tape and other processes special effects engineer should be able to utilize the user 
to complete a video production. Consequently, com- interface in a manner consistent with their work envi- 
puter controlled editing systems and integration meth- ronment, which may, by way of example, be illustrated 
ods have been used to incorporate and integrate various 20 through the use of video time code signals (and not 
production media resources such as special effects, mu- music). 

sic, graphics and the like. However, due to the nature of As wfll be described, the present invention provides a 
film and video production, a variety of resources must multi -media user interface which may be utilized by a 
be integrated, scheduled and coordinated with one an- variety of multi-media professionals in the production 
other to obtain a completed product 25 Qf fl|m m &pe works UrdU-,. prior art window based 

Historically, humans have interfaced with computers display systems, the present invention provides a three- 
through a system of discrete commands which typically dimensional representation (known as an "element") of 
comprise a combination of both text and mathematical information to be manipulated. Each element comprises 
symbolic characters Examples of such systems are ^ ^^o^ representation of a "resource." A 
numerous and include the programming languages of 3D fc & y^^^^^] which my have 

Fortran, Algol. Basic, etc., which transform 1 a pvenset repre sented in either two or three-dimensional 

of. user commands into mac^ executable "object J window comprises a "venue" which may be 

code. However, the ease with which a user becomes ^ ^ for spedl / activ j tieif 8Uch „ music pro duc- 
proficient m programming or interacting with a com- »^ efr^schedulmFand the like. However, 

puter-based system is generally a function of how close 35 UUI1 ' 'i^r—^ 7^ » e j . . , . ' 

L system modeb the logical thought of the user him- * e ™* m , tcrfa « fundamental tools and 

self If the user is able to efter commands in the order in ***** debase, such that each media professional, 
which he would find most logically appropriate, rather » ■ vtdeotapeeditor, audio editor, producer, etc 

than having to transpose his desired command into the mterface m a cons*tent manner, 

code of a programming language, greater user effi- 40 SUMMARY OF THE INVENTION 

ciency in using the system Is achieved. . _ .. ^ , . 

A number of systems which have been developed to The present invention provides apparatus and mcth- 
nnnimize the learning or acclamation period which a ods for a multt-dimensional user interface for use in 
user must go through to become proficient in the inter- audio visual production. A display system including at 
action with the computer system are referred to as "ob- 45 least one central processing unit (CPU) is coupled 
ject oriented systems". A common object oriented in- «tawgb appropriate mput/output Q/O) circuitry to 
terface approach utilizes multiple "windows" displayed devices, such as a keyboard, digital pad and/or 

on a cathode ray tube (CRT) in which combinations of tiatk ball. The CPU is further coupled to a hard disk 
text and graphics are used to convey information. Each drive for the storage of programs and data, and is also 
window may take the form of an object such as a file 50 coupled to a network through which the CPU may 
folder, different operating environment, pages or lay- communicate with a variety of system resource devices 
ered bit maps to provide a separate display of video such as editors, music synthesizers, graphics generators, 
information in independent screen regions. (See, for scheduling resources, audio enhancement resources, 
example, Robson, "Object Oriented Software Sys- etc. The CPU is also coupled to a display device (for 
terns", Byte, August, 1981; and U.S. Pat. Nos. 4,414,628; 55 example, a CRT) on which the present invention's user 
4,533,910; 4,450,442; 4,555,775 and 4,622,545, and. L. interface is displayed to the user. A user viewing the 
Tesler, "The Small Talk Environment", Byte, August, interface on the display may utilize one of the input 
1981, Volume 6, No. 8.) devices, such as by way of example, the keyboard, to 

The use of modem computer systems incorporating select, incorporate or otherwise integrate the various 
object oriented window environments may be applied 60 system resources to develop a unified multi-media pro- 
to multi-media production methods, such as videotape duction. The user interface of the present invention 
editing, audio mixing, etc. However, one unique prob- includes a control frame which in practice substantially 
Jem associated with multi-media production is the ne- fills all of the display screen of the display. The control 
cessity to provide the ability for a diversity of media frame is comprised of control panels which surround a 
professionals to collaborate and exchange project data 6S variety of subwindows, and acts as a consistent control 
in a consistent interface environment. By providing a area for all users of the interface. The control frame 
consistent user interface, media professionals such as includes a construction area which corresponds typi- 
special effects engineers, animation specialists, music caBy to a front view port looking toward a three-dimen- 



04/14/2004, EAST version: 1.4.1 



5,148,154 

3 4 

sona) element which is a representation of a resource. The manipulations performed are often referred to io 
The control frame further includes a top view port terms, such as adding or comparing, displaying, etc. 
which illustrates the time relationship between the van- which are commonly associated with mental operations 
ous resources in a "venue". Effectively, the control performed by human operator. No such capability of a 
frame of the present invention provides a two-dim en- J human operator is necessary, or desirable in most cases, 
sicmal window to selectively view a' three-dimensional in any of the operations described herein which perform 
"element". In operation, a user specifies mandatory and part of the present invention. In the present case, the 
optional attributes which an element must -have, and ' operations are machine operations. Useful machines for 
defines the element representing the resource within the " performing the operations of the present invention in- 
constroction area of the control frame. Once defined, 10 elude general purpose digital computers or other similar 
the element may be selectively "dragged" down to an devices. In all cases, there should be borne in mind the 
event horizon bar at which time, time data is displayed distinction between the method operations of operating 
in the time view port of the control frame. Using the a computer and the method of computation itself. The 
interface of the present invention, elements may be present invention relates to method steps for operating 
created, edited, bundled, integrated and rearranged 15 a computer graphics system and processing electrical or 
along the event horizon. other physical signals to generate other desired physical 

BRIEF DESCRIPTION OF THE DRAWINGS ^ prescnt ^ relalcs t0 apparatus for 

FIG. 1 is a conceptual illustration of the present in- performing these operations. This apparatus may be 
vention's integrated multi-media user interface for trite- 20 specially .constructed for the required purposes, or it 
grating and manipulating a variety of multi-media func- may comprise a general purpose computer selectively 
tions. . activated or reconfigured by a computer program 

FIG. 2 is a functional block diagram showing one stored in the computer. The algorithms and displays 
possible data processing system incorporating the presented herein are not inherently related to any par- 
teachings of the present invention. 25 ticular computer or other apparatus. Various general 

FIG. 3 is a conceptual illustration of the present in- purpose machines may be used with programs in accor- 
vention's use of venues to represent data sets of re- dance with the teachings herein, or it may prove conve- 
sources available to a user. nient to construct more specialized apparatus to per- 

FIG. 4 conceptually illustrates the use of venues and form the required method steps. The required structure 
view ports by the interface of the present invention. 30 for a variety of these Tfhinrs will appear from the 

FIG. 5 is a conceptual illustration of an element rep- ' description below. In addition, -the present invention is 
resenting a resource, as viewed in three dimensions not described with reference to any particular program- 
utilizing the present invention's user interface. ming language. It will be appreciated that a variety of 

FIG. 6 is a front view of a user display screen utiliz- programming languages may be used to implement the 
ing the present invention's multi-media user interface. 35 teachings of the invention as described herein. 

FIG. 7 is the same view as FIG. 6 except with the nFTATT pn 

ehnent attributes window opened for venue selection. DCTAIL^ D^WPTION OF THE 

FIG. 8 is the tame view as FIG. 7 further illustrating INVENTION 
the present invention's user interface in the display in a The following detailed description will be divided 
plurality of elements and venues selected for the pro- 40 into several sections. The first of these will treat a gen- 
duction of a multi-media work. era! system arrangement for generating computer 

FIG. 9 is the same view as FIG. 8 further illustrating graphics in accordance with the tunning* of the present 
the element attribute block opened and venue and re- invention. Subsequent sections will deal with such as- 
sour ce selection. pects of the present invention as the general conceptual 
xinivnrxi^t a -nine 43 definition of a "venue" and "resource", and the struc- 
NOTATION AND NOMENCLATURE tare in operation of the present invention's multi-media 

The detailed descriptions which follow are presented user interface. ' 
largely in terms of graphics interfaces, algorithms, and In addition, in the following description, numerous 
in symbolic representations of operations of data bits specific details, are set forth such as functional blocks 
within a computer display system. These algorithmic SO representing data processing devices, window configu- 
descriptions and representations are the means used by rations, etc. in order to provide a thorough undersUnd- 
those skilled in the data processing arts to most effec- ing of the present invention. However, it will be obvi- 
tively convey the substance of their work to others ous to one skilled in the art that the present invention 
skilled in the art may be practiced without these specific details. In other 

An algorithm is here, and generally, conceived to be S5 instances, well known circuits and structures are not 
a self consistent sequence of steps lekding to a desired described in detail in order not to obscure the present 
result These steps are those requiring physical manipu- invention unnecessarily. 

la tions of physical quantities. Usually, though not neces- _ „.~„ . . _ . „ . 

sarily, th« quantities take the form of dectrical or GENERAL SYSTEM CONFIGURATION 

magnetic signals capable of being stored, transferred, 60 Referring now to FIG. 1, the present invention is 
combined, compared, .and otherwise manipulated. It conceptually illustrated. As previously discussed, the 
proves convenient at times, principally for reasons of present invention provides an integrated user interface, 
common usage, to refer to these signals as bits, values, ' such that media professionals may utilize a common 
elements, symbols, icons, characters, terms, numbers, interface and integrate various production processes 
windows or the like. It should be borne in mind, how- 65 such as animation, special affects, editing, mixing and 
ever, that all of these and -similar terms are to be associ- production scheduling. Through the. use of the present 
ated with the appropriate physical quantities and are invention's user interface, a variety. of resources may be 
merely convenient labels applied to these quantities. accessed in the production of a feature film, videotape 
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or the like. Unlike prior art systems in which separate RESOURCE ACCESS THROUGH VENUES 
special effects production facilities, live action facilities RESOURCE ACCESS THROUGH VENUES 
and music score editing facilities are required to work Referring now to FIG. 3, the present invention con-, 
independently, and then be integrated at some later ceptually permits a variety of media resources such as 
date, the present invention provides a new editing eon- 5 special effects 90, editing sequencer 92 and video re- 
cept to permit an operator to interact with each of the sources 95 to be viewed by a user 100 through a "view 
resources comprising a multi-media production and port", relative in time, along an "event horizon" 104. As 
generate a final completed work. As will be described will be described, a view port provides a perspective 
more fully below, the present invention accesses, ar- view of data contained in a "venue", wherein a venue 
ranges and coordinates these various production re- 10 may include a plurality of resources such as audio re- 
sources through the use of an integrated user interface. sources 102, video resources 95, etc. 

Referring now to FIG. 2, one possible computer Referring now to FIG. 4, the present invention's 

graphic system employing the teachings of the present concept of venues is disclosed in relation to three-di- 

invention is shown. As illustrated, the graphics system mensional space. The venue concept of the present 

includes a computer 20, which comprises six major IS invention allows the user 100 to view data represented 

components. The first of these is a central processing in alternative view ports of a venue. Relationships be- 

unh (CPU) 22 which is coupled to a memory 24. CPU t ween a variety of resources axe apparently based on the 

22 and memory 24 are further coupled to a user inter- view which the user 100 chooses. As illustrated in FIG. 

face circuit 26 and a disk input/output (I/O) circuit 28, 4, a venue is a three-dimensional space in which object 

for communicating with a bard disk drive 30 for mass 20 "elements" reside. As "element" is a three dimensional 

storage of data. Computer 20 further includes a serial representation of a resource coupled to, for example, 

input/output (I/O) circuit 32 for communicating with the network 44 depicted in FIG. 2. Depending on the 

serial devices over line 34, such as by way of example a view port which is chosen, the elements identify them- 

digitizing pad 36, a keyboard 38, and a track ball input selves in various ways. For example, in FIG. 4, element 

device 40. Computer system 20 further includes a net- 25 110 may comprise a graphics resource, element 114 may 

work interface circuit 42, which is coupled to a network represent an audio resource, element 116 may represent 

44 and a gateway circuit 48, which permits the com- a textual planning resource, and element 118 may repre- 

puter system 20 to communicate with other communi- sent a music or other audio resource. Viewing a venue 

cation systems over telephone lines, optical fibers and through a top view port reveals the relative placement 

the like. 30 of the various elements in time, both the duration of 

In addition, a display monitor 30 is illustrated which time that an element represents (such as a film "take") 

is used to display the integrated user interface compris- and the relationship between the resources to a refer- 

ing the present invention, and is coupled to the user ence time, which may be a recording machine, finished 
interface circuit 26. As also illustrated, a variety of film or other multi-media piece. Viewing the right side 

resource devices such as a video tape recorder (VTR) 33 view port reveals versions of the elements, for example, 

52, music synthesizer 55, audio tape recorder 60 and audio element 118 includes two versions; namely, ver- 

producrion switcher 62 are coupled to the network sion 120 and version 122, along with the relative time 

interface circuit 42 over the network 44, through device lengths of the versions. In addition, it should be noted 

translators 64, 66, 68 and 70, respectively. that the user interface of the present invention permits 

As will be described more fully below, the present 40 the use of multiple venues, wherein some or ail of the 

invention as illustrated in FIG.. 2 permits data from venues may share the same time period, and each venue 

resources such as VTR 52, music synthesizer 55 and includes its own elements. 

audio tape recorder 60 to be coupled to. the present For sake of example, assume that the elements illus- 
invention's user interface. A user viewing the interface t rated in FIG. 4 represent resources for the production 
on display 50 may utilize one of a variety of input de- 43 of a soap opera. In this example, assume that a produc- 
vices, such as by . way of example, keyboard 38 or track tion crew photographed a variety of scenes, transferred 
ball 40 to select, incorporate and otherwise integrate the these scenes to video tape, and that these scenes corn- 
various system resources to develop a unified multi- prise scene 1, take 1, etc. 

media production. As shown in FIG. 5, viewing an element through the 
It will be appreciated that the present invention, as SO frost view port reveals the type of resource through the 
illustrated with reference to FIG. 2, is only one possible use of an icon label. For example, in the case of an audio 
embodiment of many. For example, although only one resource, the icon label may comprise a graphic rep re- 
computer 20 is shown, the present' invention may in- sanation of a musical note. In addition, the various 
elude multiple CPU's and/or computers coupled to the versions are illustrated, as in the case of element 118 in 
network 44! Each of these CPU's and/or computers 55 FIG. 4, by darkened lines traversing the longitudinal 
coupled to the network 44 may execute separate pro- length of the rectangular element image with respect to 
grams, and support different versions of the user inter- FIG. 5, the version may be activated, and thereby run, 
face of the present invention. Alternatively, it is con- by placing a cursor on the screen of display 50 over one 
tern plated that the use of multiple computers and/or of the activation buttons 130 on the element, and pro- 
CPU's may support a system of distributed computing, 60 viding an activation signal, such as for example from a 
wherein multiple versions of the present invention's "mouse", to run that particular version of the resource, 
user interface may be supported concurrently, each of Returning to the present example of a soap opera pro- 
the multiple user interfaces executing different pro- duction, versions 120 and 122 of the audio element 118 
cesses but having access to a common data base of re- as shown in FIG. 4 and versions 135 of the element 140 
sources. As such, it will be noted that the embodiment 63 as shown In FIG. 5 may comprise musical options for 
shown in FIG. 2 is a simplified embodiment for pur- use during the particular time associated by the length 
poses of illustration, and is not meant to limit the present of the elements. Similarly, if the resource comprises 
invention's utility. scenes, then, as is common in the industry, each of these 
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scenes may have associated .time codes (such as port 215 is also provided in control frame 150. As previ- 

SMPTE) which comprise the time in and out Tor each ously illustrated in FIG. 4, elements representing re- 

scene. Accordingly, by viewing the element represent- sources may be viewed and operated upon by appropri- 

ing the resource in terms of time, the time length of each atcly selecting a view point. Although the control frame 
scene would be represented on the element by the 5 150 displays objects in two dimensions, by appropri- 

length of the particular version lines, for example lines ately selecting the view port, the elements may be 

135. or alternatively, by duration lines 140 defining the viewed from all three dimensions. Elements, utilized in 

beginning and end time of each scene of the resource. (he present invention, directly represent the objects that 

INTEGRATED USER INTERFACE , n make "P 6 P"* 1 " 1 ^ 00 - ^ 85 segments of 

w video tape, score, scenes, director notes, sound tracks. 

Referring now to FIG. 6, the integrated user inter- ctc^ and are identified by an icon on the front face of 
face of the present invention will be described. As the element as previously described with respect to 
shown in FIG. 6, the present invention's user interface FIG. 5. Elements can be viewed from the front, side, or 
includes a basic control frame 150 which in practice fills f rom the top, or in multiple views. As previously dis- 
substantially all of the display screen of display 50 as 15 cussed with reference to the concept of "resources", the 
depicted in FIG. 2. The control frame 150 is. the pri- ^ ^tv? of ^ element displays the icon label and the 
mary context for interacting with the user interface of type of dement may be determined from the icon on its 
the present invention, and is, primarily, comprised of facc (see nQ j) viewing an element from the top 
four control panels which surround a variety of subwuv ^ rt mvstntes ^ relative length in time the ele- 
dows. Three of the panels are visible to the user at all 20 meDt ^ A ^ from a ^ ^ ^ illustrates 
times, and the ^bottom panel (not shown) is displayed on difrerent versions and their relative lengths. The 
an as needed basis. The mode specific acOon panel 152 element/resource may comprise several Individual ele- 
comprises, in the displayed mbodiment, a plurality of m ^ ^ bundled - mto , new compound de . 
direct acnon buttons, or to which change Wiethe ment muc]] ^ current ^ may group graphical de- 
program mode. As shown in the illustration of FIG. 6, 25 , f . „ ,_»,.■,-. «j;,„, 

^specific action pane! 152 comprise a vi^al Referring on^ag^S toFlG. 6, the top view port 215 

element 155, an aural element 158, a text element 162, a . . ,„ _Jl lti __ .iL™,. 

graphic element 164, and a process element 166. Al- 15 ."f 3 * y ** . mv f""> a 10 P?* 00D dcaiea * 

Sotgh buttons 155. 158. 16* 164, and. 166 are Olus- f^ vc ^ ^ Placing them on . tune line. As will 

tratedin FIG. 6, iTwuVbe appreciated that the mode 30 «* ^ ^ S^" 8 ff^T 

«pccif.c action panel buttons/icons change to support ° n f ^ ' to . » *f "event horizon 

andreflect the current activities of the user for a parte- grates the dement into the overall tome hne for the 

ular venue. production and results in the display of tune data m the 

The control frame 150 further includes a major mode ™» identified » <" 218. Moving jin element 
panel 170, which comprises an edit button 172. a graph- 33 frt ™ construction rc P° n » to the event horizon 220 
icTbutton 174, an audio button 176, a plan button 180, ***** m the dement bemg assigned a tone assignment 
and a set up button 182. It will be noted that although in the view port 215. It will be noted that the view port 
throughout this description icons, buttons, and the like 215 corresponds to the top view port view of an de- 
are described, that the reference to buttons, icons, etc. ment * * e three-dimenaonal representation of re- 
represents any class of displayed hems which result in 40 """f described ^ ' d j erencc to P G - 4 , 
some executable action when chosen by a user. There-. Referring now. to FIG. 7, assume for sake of example 
fore, although an edit button 172 is disclosed as part of ^ 8 nser desires to define an element using the present 
the control frame 150, it will be appreciated to one invention's user interface. An dement attribute box 250 
skilled in the art that the edit button 172 may comprise fa displayed by computer 20 depiceted in FIG. 2 once a 
an icon in the form of some edit feature or the. like 43 selection identified as "dement" 252 is chosen by a user, 
which achieves the same result In the presently pre- Although in the presently preferred embodiment, de- 
ferred embodiment, the buttons comprising the major ment 252 comprises a pull down menu (not shown) 
mode panel 170 are always present for the user no mat- having a plurality of items for sdection, conceptually 
ter which venue or other option is selected. the dement 252 is selected by a user through the use of 

In general, the major mode panel 170 permits a user 30 • amot control device (see for example, U.S. Pat No. 

to access different venues than the venue currently RE32633). Although a cursor control device, such as a. 

displayed. The specific buttons/icons used in the major "mouse" may be used to select dement 252, it will be 

mode pand 170 is a function of the particular project in appreciated that the actual sdection may be made using 

which the user interface is implemented. A menu bar » variety of display input devices known in the art It 

panel 190 generally displays labels for pull down menus. 33 will also be appreciated by one skilled in the art, that 

Standard labels such as "organize", "modify", etc. are other mechanisms for selecting dement 252 and the like 

provided and are present no matter which venue or are known. Accordingly, the particular mechanism for 

resource is accessed. Other context specific menu labels sdecting functions, items, icons and the like within 

will be displayed such as "element", "edit style", "utili- control frame 150 will not be described in detail in this 

ties", "setup", etc. are provided for specific applies. 60 Specification. 

tions. Once dement 252 is sdected, computer 20 illustrated 
The application specific construction area 200 com- in FIG. 2 displays the dement attribute box 250 as 
prises an application area of the display for a selected shown in FIG. 7. A user then either sdects from a pre- 
program mode, and is available for subwindows, user printed list of dements, or defines dements within the 
views in the display of other elements of work such as 65 element attribute box 250, the nature of the resource 
time lines, scripts, scores, preview monitors, etc. As required for the particular production. Examples of 
shown in FIG. 6, construction area 200 is designated as such element attributes which a user may select include, 
a resource manager. As will be described, a top view but are not limited to the following: 
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VISUAL ELEMENT ATTRIBUTES 
Label 



User supplied Element identification, (e.g.: Video i 
Tape Recorder, etc) 

Source Device 

Assignment of device: (eg.: PI, P2, RECORD, etc.) 

Identification 



TEXT ELEMENT ATTRIBUTES 
Label 

User supplied Element identification. 

Name 

Title of Text type and Document. (SCRIPT. OPEN- 
ING TITLE, CHARACTER GENERATOR, ETC) 



10 



Alpha-Numeric material Identification: e.g.: reel 
number, reel label, etc. lab roll number, etc. 

Scene/Take Information 

Content Scene and Take identification. 



15 



Scene Take file name 



Codes 



Time Code and type. Origin Time Code and type, 
User bit Time Code. User bits. Content. Frame Num- 
bers. Edge Numbers. Code Numbers. 

Notes 

Associated Text Notes For Reel &/or Content. 
Priority 

User assigned Priority levels for different versions. 
Processing Path 

Information on any previous Processing that applies 
to this material, (eg.: DUB LEVEL, COLOR COR- 
RECTION, ETC.) 

AUDIO ELEMENT ATTRIBUTES 

Label 

User supplied Element identification. 

Source Device 

Type and Assignment of device, (e.g. ATR, DISC, 
ETC) PI, P2, RECORD. ETC 

Identification 

Alpha-Numeric material identification, (e.g. Reel 
number, Reel label etc. Sound Roll Number, Label.) 

Scene/Take # 

Content Scene and Take identification. 

Codes 

Time Code and type. Origin Time Code and type. 
User bit Time Code contents. Frame Numbers. 

Tracks 

Number and Numbers of Source Track 
Notes 

Associated Text Notes. For Reel &/or Content 
Priority 

User assigned Priority levels for different versions. 
Processing Path 

Information on any previous Processing thai applies 
to this material, (eg. Dub level, Equalization, etc.) 



20 



25 



30 



35 



40 



Revision- 

The current Text revision level relevant previous 
revision information. 

File Types 

The Names and Types of files as the material exists in 
useable form. 

Associated Data 

Any previous data files associated with creating the 
current file. 

Processing Path 

Information on any previous Processing that applied 
to this material 

GRAPHIC ELEMENT ATTRIBUTES 
Label 

User supplied Element identification. 

Title 

A user supplied description of the Graphic element 
Revision 

The current Graphic revision level and relevant pre- 
vious revision information. 

File Types 

The Names and Types of files as the material exists in 
useable form now. 



45 



SO 



55 



60 



65 



Associated Data 

Any previous data files associated with creating the 
current file. 

Processing Path 

Information on any previous Processing that applies 
to this material. 

PROCESSING ELEMENT ATTRIBUTES 
Label 

User supplied Element identification. 

Device Identification 

Effects Device identification. 

Assignment Path 

Video and/or Audio routing assignments, (e.g. 
CROSSPOINTS, KEY CHANNELS, ETC.) 

Up Load/Down Load 

File input/output for created effects save and recall. 

Codes 

Time Line Code and type. Effects durations. Effects 
Source codes. Effects Edge Numbers for optical printer 
outputs. 
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p dement may be changed by simply "dragging" an ele- 

processing ment such as scene I 310 off of the event horizon 220, 

Effects types. (e.g. CUTS, FADES, DISSOLVES, moving other dements into that time slot along the 
WIPES, KEYS, DME, ETC.) event horizon, and replacing scene I 310 at some other 

Once the dement attributes have been defined,. com- 5 location along the event horizon. A redistribution of the 
puter 20 illustrate in FIG. 2 utilizes appropriate network element's relative position along the event horizon 
connections over network 44 to the various resources, would correspond in FIG. 4 to, for example, swapping 
such as the VTR 52, music synthesizer 55, Audio tape dement 110 for dement lit! and vice versa, 
recorder 60, Production switcher 62, etc. to access the Once an clement is placed upon the event horizon 
resource via the user interface. Accordingly, a direct 10 220, position data relative in time to other dements is 
connection via computer 20 has been created between illustrated along the time lines of the top view port 215 
the user interface comprising the control frame 150 as as shown. Conceptually, the reader is directed to FIG. 
displayed on the display 50, and the particular dement- 4 which illustrates in three dimensions the placement of 
/resource coupled through the network interface 42. dements relative to one another in time. However, due 
Refering to FIG. 7, within the top view port 215, time 15 to the limitations of display 50, depicted in FIG. 2 the 
dements corresponding to the particular resource have time view port 215 is utilized to display time along the 
additional information that is revealed in the time line ± Y direction, with time T« being at the lower portion 
associated with the top view port 215. Tracking buses of the display as illustrated in FIG. 8. In addition, as 
265 provide additional information regarding the re- shown in FIG. 8, the number of versions of, for exam- 
cording of audio channels 1 through 4, and a video 20 pie, scene I 310, is also displayed as versions 351 
channd 1. In practice, a source tape msdiitw (not through 354. It win be appreciated by the reader that 
shown) supplies audio to a tape record machine wherein the display of an dement such as scene I 310 corre- 
the channels are coupled to one another. It has been sponds to the prior description of a resource having 
found that it is quite useful to display audio channels in multiple versions which may be activated by selecting 
the time view port 215, in order to correlate the audio 23 (for example, by placing a cursor over version 354 and 
channd and time interval versus resource. providing an activation signal), a version such that the 

Once the element attributes have been defined, an version is "run" within the window of scene I 310. 
dement representing a resource is created based upon Accordingly, user may view the entire version of scene 
those attributes, and displayed within construction area I which has been sdected, within the icon window 
200 of the control frame 150. Referring now to FIG. 8, 30 comprising the scene. In general, in the present and 
the present invention's user interface is illustrated preferred embodiment, and throughout this specifica- 
wheretn a plurality of dements identified as Record tion, placing a cursor over an icon or other executable 
VTRI 300, Scene I 310, scene II 312, Dissolve 314, function and "double clicking" using a cursor control 
Open Title 316, and Key 320 are shown. As illustrated - device such that two consecutive signals are provided 
in FIG. 8, an element such as Record VTRI 300 in- 35 to the computer 20 depicted in FIG. 2 executes the 
dudes an icon image (for example, the planet Earth in function which has been sdected, and, more particu- 
FIG. 8) which describes some aspect of the element for larly, reveals any attributes and/or contents of the icon, 
identification purposes. Viewing dements disposed in Double clicking on a time function such as time block 
construction area 200 normally corresponds to viewing 400 for scene I (FIG. 8) may be configured such that 
a venue and associated resources through a front view 40 time code (i.e. SMPTE) is displayed. More particularly, 
port as shown previously with respect to FIG. 4. An in the present example, SMPTE time code for the be- 
dement, for example, Record VTR 300, may be moved ginning and end of each version within scene 1 310 may 
within the construction region 200 at will by a user be displayed within the time line view port 215. 
through the use of an appropriate command sequence, Referring now to FIG. 9, control frame 150 is 01ns- 
or by simply dragging the dement around the construe- 45 trated in which a new scene 1 400 has been "dragged" 
tion area using a cursor control device such as a mouse. upward into the construction area 200. As Illustrated in 
However, using the teachings of the present invention, FIG. 9, once an dement, in the present example scene I 
once an dement such as Scene I 310 is brought,' 400, is moved off of the event horizon 220, timing infor- 
dragged, or otherwise manipulated downward to the mation viewed through top view port 215 corres pond- 
event horizon 220, the dement is automatically given 30 ing to scene 1400 is no longer displayed. Elements such 
time significance which is represented along the time as scene II 404, dissolve 406, open title 408 or key 410 
hnes of the top view port 215. may be repositioned along the event horizon 220 and/or 

As illustrated in FIG. 8, the event horizon 220 com- modified in terms of time sequence as viewed through 
prises a horizontal bar with arrows 221 and 222 at each . top view port 215, relative to one another. Alterna- 
of its opposite ends. By placing a cursor (not shown) SS tivdy, and as shown in FIG. 9, dement attribute box 
over arrows 221 or 220, and presenting computer 20 250 may be sdected and the attributes of scene I '400 
with an activation signal, resource dements such as ' may be modified, or an entirely new element may be 
scene I 310, scene II 312, Dissolve 314, etc. may be defined, to replace scene 1 400 along the event horizon 
moved left or right, respectively, and other dements 220. 

may be viewed which are currently not visible on the 60 It will be appreciated that a user utilizing the inter- 
event horizon in the control frame 150. The use of ax- face comprising the present invention defines elements 
rows 221 and 222 permits a user to scan through de- in the construction area 200 by specifying attributes of 
ments disposed on the event horizon 220 and view the the dement in the dement box 250. In addition, multiple 
dements not only in relative position, but in rdative dements may be created within the construction area 
time. This view corresponds to that of a user 100 in 65 200. The created dements may then be selectively 
FIG. 3 scanning the resources in that Figure, such as dragged to the event horizon 220 in an arrangement and 
special effects 90, editing 92, video 95, etc. In addition, order selected by the user. It will further be appreciated 
it will be appreciated that the relative position of the from the above discussion by one skilled in the art that 
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the present invention's user interface permits the utiliza- which a user interacts through said user input 

tion of resources within the system illustrated in FIG. 2, means; 

permits selective modification of the resources, and said representations of said resources comprising. at 

through the use of the interface of the present invention least three dimensions and being arranged in a 

provides a consistent interface for the production of an ' 5 venue of at least three dimensions, each of said 

audio visual work. The user interface of the present dimensions corresponding to an attribute of said 

invention, through the use of a common control frame resources, said venue may be viewed using said 

150 as displayed an display 50, allows artists, musicians, interface from a plurality of view ports, such that 

and other media professionals to create, modify and viewing said representations of said resources from 

rearrange resources comprising a production with flexi- 10 different view ports results in the display of differ- 

bility heretofore unknown in the art The present inven- attributes of said resource s; 

lion's concept of venues, and the ability to operate on s^d representations of said resources being arranged 

resources in three dimensions, provides a user with in said venue such that each of said representations 

flexibility not present in any prior user interface for » disposed relative to one another in time and 

computer display systems, as well as multi-media pro- 15 s P ace , u ? thm sajd venue J 

duction systems known in the past manipulation means coupled to said user input means 

The present invention has been described and illus- for selectively positioning said representations of 

trated with reference to the Figures as applied to a , said. resources within said venue, 

display 50, and using input devices, such as digital pad ,„ * "defined by chum 1. wherein 

36. trackball 40 mri keyboard 38 as shown in FIG I md venuc " ^ dimensions and each of aid repre- 

However, the teachings of the present invention may be mentations of said resources comprise three pensions, 

applied to numerous other display devices and input * The d^Uysys em as defined by claim 2 wherein 

Hh ' . _ i .v . • • said venue may be viewed from six view ports, namely, 

mecharasnu. For example, the present invention may be a top, left* right. Tear, bottom and front view port, 

practiced usmg what is knownas '-virtual reality" mput ^ ^ ^ m ^ defmcd b claim 3 ^ herein 

devices such as but not limited to, a data mput glove, ^ vcnue from , t ^ po n reveals ^ 

body glove mput device, etc. In addition, the present relative positions ofcach oTsa id three dimensional rep- 

mvenaon may be utilized with eye goggle displays resenUuions of said resources relative in time to one 

which are worn by a user and coupled to the computer another 

display system via fiber optics, wires and the like. When M 5 ^ system ^ defuMd by 4 wherein 
the present invention is utibzed in conjunction with a yj^g vcnue f rom a f ront view port reveals an 
virtual reality system, the user interface of the present icon identifying the type of resource each of said rep re- 
invention would be viewed by a user through input sentatians represent 

goggles as being suspended in space. Interaction with £ The display system as defined by claim 5 wherein 

the interface by the user may be done using an input 35 viewing said venue from a side view port reveals ver- 

glove or other virtual reality device worn by the user. siQDi of resource m <j the lengths of said versions 

Accordingly, it will be appreciated that the user inter- relative to one another. 

face of the present invention is not limited to conven- 7. The display system as defined by claim 6 wherein 

tional input or display devices. The reader is referred to viewing said venue from a front view port further re- 

the following references for a further description of 40 V eals version activation buttons, one of said buttons for 

existing and proposed virtual reality systems. Computer- eic ] 1 0 f said versions, such that said user, using said user 

iied Reality Comes of Age, NASA Tech Briefs, page 10, input means may selectively activate said versions. 

August 1990 (Vol. 14, number 8); Iwata, Artificial Real- g. The display system as defined by claim 7 wherein 

Ity with Force— Feedback Development of Desktop Vir- m fa e event said user activates one of. said versions, said 

tual Space with Compact Master Manipulator, ACM 4$ version is executed within said representation of said 

SIGGRAPH, August 1990 (Vol. 24, number 4); Nash, resource, and may viewed through said front view port 

Our Man In Cyberspace Checks out Virtual Reality, Com- 0 f said venue on said display, 

puterworld, Oct 15, 1990; Da visa. Grand Illusions, 9. The. display system as defined by claim 6 wherein 

Discover, June 1990. said user interface includes interface generation means 

White the present invention has been described in 30 coupled to said CPU for generating and displaying a 

conjunction with a few specific embodiments identified control frame using said display means to display se- 

in FIGS. 1 through 9, h will be apparent to those skilled lected view pons of a venue, said control frame includ- 

in the art that many alternatives, modifications and ing a plurality of command options which may be sc- 

variations in light of the foregoing description are possi- lected by said user using said user mput means, 

ble. Accordingly, the invention is intended to embrace 55 10. The display system as defined by claim 9 wherein 

all such alternatives, modifications and variations as said control frame further includes a first area for defin- 

may fall within the spirit and scope of the invention as ing said attributes of said resources and displaying said 

disclosed. representations of said resources once said attributes are 

We claim: defined. 

1. A computer controlled display system including at 60 11. The display system as defined by claim 10 

least one Central Processing Unit (CPU), said CPU wherein said first area displays said representations of 

coupled to a display for displaying data, and user input said resources in a venue initially from a first view port, 

means, said display system further coupled to a plurality 12. The display system as defined by - claim 11 

of system resources having defined attributes, said dis- wherein said user may selectively change view ports 

play system comprising: 6J from said first view port by selecting one of said com- 

a user interface for display by said display means, said maud options, 

user interface for displaying representations of 13. The display system as defined by claim 10 

resources coupled to said display system with wherein said control frame further includes a second 
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area for displaying said top view port of said venue, 23. The display system as defined by claim 22 

such that timing data representing relative time associ wherein said command options of said control frame 

ated with said resources is displayed in said second area. further includes a plurality of mode specific action op- 

14. The display system as defined by claim 13 tions on the periphery of said control frame, 
wherein said control frame further includes a event 5 24. The display system as defined by claim 23 
horizon bar, such that the placement of a representation wherein said representations of said resources are ar- 
of a resource on said bar results in said timing data being ranged in said venue such that each of said resources is 
displayed in said second area of said control frame. disposed relative to one another in time and space 

15. The display system as defined by claim 14 within said venue. 

wherein selecting an Element command option results 1° 25, The display system as defined by claim 24 further 

in the display of an element attribute box for defining " including manipulation means coupled to said user input 

said resource attributes. means for selectively positioning said representations of 

16. The display system as defined by claim 15 &»id resources within said venue. 

wherein said representations of said resources may be 26. The display system as defined by claim 24 

selectively placed on said event horizon bar, thereby " wherein said venue is three dimensions and each of said 

altering the relative placement of said representations in representations of said resources comprise three dtmen- 

said venue in time and space. sions. 

17. The display system as defined by claim 16 27. The display system as defined by claim 26 
wherein said command options of said control frame wherein said venue may be viewed from six viewports, 
further includes a plurality of mode specific action op- namely, a top, left, right, rear, bottom and front view 
tions on the periphery of said control frame. 

18. The display system as defined by claim 17 28. The display system as defined by claim 27 
wherein said command options of said control frame wherein viewing said venue from a top view port re- 
further includes a plurality of major mode options on 2J veals the relative positions of each of said three dhnen- 
the periphery of said control frame. sional representations of said resources relative in time 

19. The display system as defined by claim 18 t0 one another. 

wherein said first view port initially displayed com- 29, Th e display system as defined by claim 28 

prises the front view port wherein viewing said venue from a front view port 

20. A computer controlled display system including M revea]s w icon identifying the type of resource each of 
at least one Central Processing Unit (CPU), said CPU ^ representations represent . 

coupled to display means for displaying data on a screen 30. The display system as defined by claim 29 

of a display, and user input means, said display system wherein vising ^ venue from a side view port re- 

further coupled to a plurality of system resources hav- vea]e versjont of ^ resource and the lengths or said 

ing defined attributes, said I display system comprising: 35 vereio[ls felative to 0De ^,0^. 

a user interface on said screen of sad display for 31 The system as defined by claim 30 

displaying representations of resources coupled to wherein ^ venue from a ^ ^ ^ 

said d*play system with which a user interacts further revea]s version activation buttons , one 0 f£i d 

through said user input means; buttons for each of said versions, such that said user, 

sad user interface including mterface generation «, ^ meaM m v selectivdy activate said 

means coupled to said CPU for generating and versions. 

displaying a control frame on said display, said _. ' „ , _ . _ . . , ■ ,. 

Ztrolframe including a pluraUty of command «■ 'J* **pl*y system as defined by chum 31 

options which may be selected by said user using wheran m * e evcnl ^ ^ vates J 0Ile of ^ 

said user input means, and further including a even! 45 vers,0 f- veraon * ***** wrtrnn smd representa- 

horizon bar, such that the placement of a represen- ? on of . ^ ou & h 

tation of a resource on said bar using said user input front v^ po^rt of said venue <m said display 

means results in predetermined, data being dis- 33. The display system as defined by claim 32 

played in said control frame and said represent*- whcrein ^ «"» W*** said representations of 

tions of said resources being disposed relative to M Mld "sources in a venue imUally from a first view port 

one another in time; 34 Tne display system as defined by claim 33 

said representations of said resources comprising at wherein said user may selectively change view ports 

least three dimensions and being arranged in a from P° rt b V selecting one of said com- 

venue of at least three dimensions which may be m * n(1 options. 

viewed using said user interface from a plurality of 55 35 The display system as defined by claim 34 

view ports, such that viewing said representations ' wherein said control frame further includes a second 

of said resources from different view ports results "ea for displaying said top view port of said venue, 

in the display on said screen of different attributes such that timing data representing relative time associ- 

of said resources. ated with said resources is displayed in said second area. 

21. The display system as defined by claim 20 60 36. The display system as defined by claim 35 said 
wherein said control frame further includes a first area representations of said resources may be selectively 
for defining said attributes of said resources and display- placed on said event horizon, thereby altering the rela- 
ing representations of said resources once said attributes tive placement of said representations in said venue in 
are defined. time and space. 

22. The display system as defined by claim 21 65 37. The display system as defined by claim 36 
wherein selecting an Element command option results wherein said command options of said control frame 
in the display of an element attribute box for defining further includes a plurality of major mode options on 
said resource attributes. - the periphery of said control frame. 
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38. The display, system as defined by claim 37 44- The method as defined by claim 43 wherein said 
wherein said first view port initially displayed com- venue is three dimensions and each of said rcprcscnta- 
prises the front view port tions °* resources comprise three dimensions. 

39. In a computer controlled display system including 45. The method as defmed by claim 44 further tnclud- 
at least one Central Processing Unit (CPU), said CPU 5 ^ ? te P of «*«*™ely viewing said venue from one 
coupled to a display for displaying data on a screen of of « ™P°™> « W- nght. rear, bottom 

. ■ . . ■ . -j j- i _ Md front view P°t 

said display, and user input means, said display system ^ ^ defined b c!ainJ 45 whcrein vj ew . 

further coupled to a plurality of system resources hav- mg ^ VMUe from , top ^ ^ reveals ^ relative 
ing defined attributes, a method for manipulating and 10 portions Q f of said three dimensional representa- 
displaying representations of said resources on said tions of said resources relative in time to one another, 
display, comprising the steps of: 47. The method as defined by claim 46 wherein view- 
providing a user interface on said screen of said dis- ing said venue from a front view port reveals an icon 

play for displaying representations of resources identifying the type of resource each of said representa- 

coupled to said display system with which a user 13 tions represent 

interacts through said user input means; 48. The method as defined by claim 47 wherein view- 
generating and displaying a control frame on said m « venue from » side view port reveals versions of 
display using interface generation means coupled "3°™* thc len « ths of verrions nJative to 

to said CPU said control frame mdutog aplural- ^ ^l^^od as defined by claim 48 wherein view- 

rty of command options which may be seized by fc ^ ^ {[Qm fl frQnt ^ 

said user usmg said user input means, and further versjon ^v^o,, buttons, one of said buttons for each 

including a event horizon bar, such that the place- of versions, such that said user, using said user input 

ment of a representation of a resource on said bar means may selectively activate said versions, 

using said user input means results in predeter- 25 50. The method as defined by claim 49 further includ- 

mined data being displayed in said control frame, ing the step of said user activating one of said versions, 

and said representations of said resources being said version then being executed within said representa- 

disposed relative to one another in time; tion of said resource, viewed through said front view 

said representations of said resources comprising at port of said venue on said display, 

least three dimensions and being arranged in a 30 51. The method as defined by claim 50 wherein said 

venue comprising at least three dimensions which ■*» ^P 3 ?** representations of said resources 

mav be viewed usino said user interface from a m a venue initially from a first view port. 

. .. - . . . . . 52. The method as defined by claim 51 further includ- 

plurality of view ports, such that viewing said rep- . . , ., u"u"-u^j ~* '»""" *"" uu 

F * , r; ' , 7„ • mg the step of said user selectively changing view ports 

resentations of said resources from different view 3J ^ ^ ^ ^ by one of ^ d ^ 

ports results m the display on said screen of differ- awnd options. 

ent attributes of said resources. 53. Trie method as defined by claim 52 wherein said 

40. The method as defined by claim 39 wherein said control frame further includes a second area for display- 
control frame further includes a first area for defining ing said top view port of said venue, such that timing, 
said attributes of said resources and displaying represen- 40 data representing relative time as s oc i a t ed with said 
tations of said resources once said attributes are defined. resources is displayed in said second area. 

4t The method as defined by claim 40 further includ- 54- The method as defined by claim 53 further includ- 
ing the step of selecting an Element command option m K 106 ste P of selectively placing said representations of 
and displaying an element attribute bo* for defining said resources on said event horizon, thereby altering 
resource attributes * 5 the relative placement of said representations in said 

41 The method as defined by claim 41 wherein said v " uc Jf spa f r . . . . _, . . '.. 

. . ... , , . . _ , _,,.,„ 55. The method as denned by claim 54 wherein said 

wmmand options of said control ^ frame further includes m|n!ld options of said control frame further includes 

a plurality of mode specific action options on the pc- a plunjJitv of options on ^ periphery of 

riphery of said control frame. 50 said control frame. 

43. The method as defined by claim 42 wherein said S6. The method as defined by claim 55 wherein said 

representations of said resources are arranged in said first view port initially displayed comprises the front 

venue such that each of said resources is disposed rela- view port 

tive to one another in time and space within said venue. • • * • • 

S5 



60 



65 



04/14/2004, EAST Version: 1.4.1 



